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ABSTRACT 


The use of the Weibull failure distribution model has 
proven valuable in reliability analysis within the 
aeronautics industry. The Weibull analysis of samples with 
a high percentage of n o n - f a i 1 u r e s , or censored 
observations, must be undertaken using one of several 
large-sample approximations to the distributions of 
parameter estimators, since exact methods for such 
distributions are mathematically intractable. It is 
unknown whether these approximations will produce 
satisfactory results when used with samples, typical of the 
space shuttle main engine data, which contain small (fewer 
than 15) numbers of failures. An objective of this study 
was to design and implement a Monte Carlo computer 
simulation to assess the usefulness of the Weibull methods 
for such sm al 1 - f a il ur e samples. In particular, under 
varying assumptions concerning failure distribution 
parameters and number of failures in the sample being 
analyzed, approximate 90J confidence intervals for 
predicted times to failure for different percentages of 
components are calculated by numerical methods. The 
confidence coefficient for these intervals is then tested 
by determining what percentages of intervals trap the true 
values of the parameters being estimated. Also, 
modifications to a Weibull analysis computer program 
incorporating methods for calculating approximate 
confidence intervals and including a number of options for 
analyzing interval data (as opposed to point data) are 
descr ibed . 



INTRODUCTION 


The Weibull probability distribution has become a widely 
used lifetime distribution model since its introduction in 
1951 (3). It has been found to be especially important in 
reliability analysis of manufactured items. Although there is 
a large body of literature on the distribution and its 
statistical properties, the distributions of many of the usual 
parameter estimators seem to be mathematically intractable. 
The mathematical difficulties encountered in analyzing these 
estimators are compounded when estimating with samples that 
include data which has been censored in nontrivial ways. The 
usual methods employed in this situation involve the use of 
some statistics whose asymptotic behaviors are understood, but 
whose applicability for small samples is unknown or unsure. 

More particularly, Weibull analysis has proven to be 
quite valuable in reliability studies for aircraft engine 
components (1). This success has encouraged the exploration 
at Marshall Space Flight Center of the applicability of 
Weibull techniques in reliability analysis for components of 
the space shuttle main engine (SSME). A primary restriction 
for the SSME environment is a severe limitation on the test 
sample size. Test and flight results are collected from fewer 
than 30 engines. For major SSME components, data samples may 
contain fewer than 5 failures. 


This report documents the incorporation of a number of 
additional capabilities into an existing Weibull analysis 
computer program and the results of a Monte Carlo computer 
simulation study to evaluate the usefulness of the Weibull 
methods using samples with a very small number of failures and 
extensive censoring. Since the censoring mechanism inherent 
in the SSME data is hard to analyze, it was decided to use a 
random censoring model, generating censoring times from a 
uniform probability distribution. 

Section 1 of the report describes some of the statistical 
techniques and computer programs that are used in the SSME 
Weibull analysis. The methods documented in (1) were 
supplemented by adding computer calculations of approximate 
(using iterative methods) confidence intervals for several 
parameters of interest. These calculations are based on a 
likelihood ratio statistic which is asymptotically a chi- 
squared statistic with one degree of freedom, the basic method 
being taken from (2). 

The assumptions built into the computer simulations are 
described in section 2. The simulation program and the 
techniques used in it are described there also. Simulation 
results are tabulated for various combinations of Weibull 
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shape parameters and the numbers of failures in the samples, 
in the chart of section 3» Conclusions concerning the 
validity of the chosen Weibull model and estimators, for the 
small samples characteristic of the SSME reliability analysis, 
are drawn from the simulation results. 

In section 4, some implications of working with interval 
data, as opposed to point data, are explored. Modifications 
to the Weibull analysis computer program, with several options 
for handling interval data are described briefly. Finally, in 
the final section, some conclusions concerning the 
applicability of the Weibull methods to the current SSME 
hardware analysis are reviewed and summarized. 


1. DESCRIPTION OF STATISTICAL MODEL. 

A brief description of the statistical model and methods 
used in the SSME reliability analysis is given in this 
section. For a more complete discussion of these methods and 
tools the reader should consult references (1) and (2). The 
Weibull distribution used is the two parameter Weibull whose 
probability density function is: 

f(t) = ^p(t\t) exp(-Ctvt) ) t>0 . 

The parameter p is called the shape parameter of the 
distribution. For many applications "^-values in the range .5 
to 3*5 are reasonable. When f3 =1, the Weibull reduces to the 
standard negative exponential distribution. The parameter ft. 
is called the scale parameter of the distribution and changes 
in simply change the horizontal scale and do not alter the 
basic shape of f(t). It is easily shown that for any Weibull 
distribution, no matter what the value of p , Pr(t < Vv ) is 
equal to 1- e"* (approximately .632). For this reason is 
sometimes called the characteristic life of the corresponding 
family of Weibull distributions. Some typical Weibull 
probability density functions are illustrated in figure 1.1. 
For convenience, ft. is taken to be 1 for each curve in that 
figure . 



0 = 30 



FIGURE 1.1 


The basic technique in the SSME use of. the Weibull 
distribution is to estimate the parameters p and K. , and 
then calculate survival times for various percentages of 
components. For example, t.i (=estimated time for 10% 
failures) would be calculated by solving F(t)=.1, where F(t) 
is the cumulative distribution function approximated by using 
the estimates calculated for p and ^ . In particular, F(t) 
is given by: 

P 

F(t) = 1 - exp(-(|^t) ) t>0. 

Two techniques are used to estimate p and h. . A 
ranked regression model produces estimates for p and Vv and 
provides for a simple graphical evaluation of whether the data 
seems to fit the Weibull model. Graphical estimation of the 
tp values can then be made. A second technique is to 
calculate maximum likelihood estimates , say p and ft , for p 
and K. and then estimate the tp using F(t) as indicated 
above. Both of these techniques are included in a computer 
program given in (1). 

It was deemed desirable to modify the above referenced 
program so that confidence intervals for the maximum 
likelihood estimators could be calculated. The type of 
censoring present in the data makes the calculation of exact 
confidence intervals impossible. Censored (or suspended) 
times can be observed for any values of t, and the censoring 
pattern is difficult to predict. The primary reason for this 
un pr ed ic t ib il i t y is that times may be censored for a large 
variety of reasons, including failures of other engine 
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components which would terminate a test. Additionally, there 
is considerable modularity built into the SSME and as a 
consequence, times at risk.for a given type of component are 
likely to be very variable from particular component to 
component. In the statistical analysis and simulation 
studies, type I (see (2) for a discussion of various types of 
censoring) censoring was assumed. Basically this assumption 
independently assigns to each component both a failure and a 
censoring time. The time observed is then the smaller of the 
two times. 

There are two standard methods which can be used to 
calculate approximate confidence intervals for the Weibull 
paramters mentioned above. For large samples (30 or more 
failures), ( ^ , ) has an approximate bivariate normal 
distribution whose covariance matrix can be estimated in a 
straightforward manner. The approximations involved in this 
method are not generally adequate for samples with fewer than 
30 failures (see (2)), and thus the method is not appropriate 
for the SSME analysis. 


A second method which seems to be a better choice for 
moderately sized samples (around 20 failures) uses a 
likelihood ratio statistic. This method has been incorporated 
into the computer program being used at MSFC. Suppose the 
sample consists of observations x, ,x t ,x 3 , ... , x^ , F= set of 
indices j, for which x_j is a failure (as opposed to a 
censored time) and r= number of failures in the sample. The 
method for calculating confidence intervals for tp is based 
on the fact that under the assumption that t p = t* , the 
statistic S, (t*) has approximately a chi-squared distribution 
with one degree of freedom. S < (t*') is given by: 

S,(t* ) = -2 log L( p , 7\ ) + 2 log L( p“ , K ). 

Here log L is the log likelihood function given by: 

log L( <3 , n ) = r log ^ -rplog K + (p-1) £ log x*-£(x*/ h )^ 

jfeF J l-\ 

A /N 

Also p and W. are the maximum likelihood estimators 
calculated from the given sample, and p is calculated 
(approximated) by solving the following equation iteratively: 

r/j -r log t* + £ log xj +log(1-p) £ ( x» / t*) P 1 og( x. / 1*")= 0. 

■* l '* i/g 

After p is found then yv =t /(-log( 1 -p)) /p 

In the computer program, is calculated using a hybrid 

secant/ fal se-position method. A ^-confidence interval for tp 
is found by finding the set of values t"* for which 
This is accomplished in the computer program by starting with 
the point estimate for tp (based on the estimates £ and ft ), 
and testing values for t ** diverging from tp in both 
directions until values are found on either side of t p for 
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which Sj exceeds the appropriate critical value. 

In a similar manner, a ^-confidence interval for p can 
be calculated by finding all values ^*for which S 2 <^^. Here 

S z ( i*') = -2 log L( p* , £ ) + 2 log L( p , K ) 

where log L, and are as before and 

** } '/P* 

Representative output from the computer program is given in 
figure 1.2. 


M- 

<.r 

L-t 


P. 

V‘ 


THE FOLLOUING ESTIMATES ARE RANKED REGRESS IONESTIMATES 


448.33 


BETA- 1.139? ETA- 

R“ 0.96470R**2- 0.93065 

DO YOU UISH TO 00 MAXIMUM LIKELIHOOD ESTIMATION? 
ANSWER Y OR N 
Y 


DO YOU UISH 80* OR 90* CONFIDENCE INTERVALS? 

TYPE IN 88 OR 90. 

90 

MAXIMJM LIKELIHOOD ESTIMATES FOR THIS CASE FOLLOW 
BETA- 1.3264 ETA- 440.80 


90* BETA CONFIDENCE LIMITS 0.83 ... 1.93 


PERCENTAGE OF FAILURES ESTIMATED TIIC 90* CONF. IN1ERUAL 


0.1 

2. 

0. 

— 

12. 

1.0 

14. 

a: 

— 

41. 

10.0 

81. 

32. 

— 

154. 

20.0 

142. 

71. 

— 

249. 

50.0 

334. 

201. 

— 

602. 

63.2 ETA 

441. 

286. 

— 

859. 

90.0 

827. 

496. 

— 

2025. 


FIGURE 1.2 


A primary problem with the above analysis in the SSME 
case is that the methods for calculating the approximate 
confidence intervals are based on asymptotic distribution 
approximations, whose validities are in question for such 
small samples. The problem is a serious one since the small 
size of the samples make us intuitively skeptical of the point 
estimates calculated from them and hence anxious to have some 
reasonably accurate confidence measure for those estimates. 
In order to address this problem, Monte Carlo experiments were 
designed and conducted to assess the accuracy, and hence 
usefulness, of the confidence intervals being calculated. 



2. SIMULATION SAMPLE GENERATION. 


Failure times were calculated by generating a random 
number r from the uniform distribution on Co.O (this was done 
using RAN, the built in random number function in the VAX 
FORTRAN), and then computing F"'(r), where F(t) is the Weibull 
cumulative distribution function for a given choice of 
parameters p and . In fact, K- was always taken to be 100 
and p was varied. Sample failure times thus generated were 
tested, using the standard chi-squared goodness-of-f it test, 
against the given Weibull distribution and found to be 
representative of that distribution. Censoring times were 
generated to follow a uniform distribution. 

To generate a sample of times, the number of failure 
times to be present in the sample was fixed. Failure times 
and censoring times were generated in pairs and the 
corresponding observed time was taken to be the minimum of the 
pair. Since the number of failures was predetermined for a 
given simulation run, the overall sample sizes varied. The 
censoring distribution range was varied (ranges to be used 
were determined by simulation) so as to achieve 
(approximately) the desired average sample size. An average 
sample size of 50 was chosen to be representative of a number 
of projected SSME sample sizes. Additional Monte Carlo 
studies, in which both sample sizes and censoring 
distributions will be varied, are planned. 

For an occasional sample one or more of the iterative 
numerical methods employed in the confidence interval 
calculations failed to converge properly. Such samples were 
omitted in the subsequent Monte Carlo analysis. The number of 
samples for which this occured was small (on the order of 1$), 
and so it is clear that ignoring them could not unduly bias 
the simulation results. At any rate, when such samples occur 
in practice, they would not be evaluated using the confidence 
interval calculations. 


3. SIMULATION RESULTS. 


Most of the Monte Carlo simulations that have been run 
are summarized in Table 3.1. Some additional isolated runs 
with varying sample sizes and censoring distributions have 
been made, but these were not systematic enough to include 
here. It might be noted, however, that these runs showed 
results consistent with the results reported here. A more 
thorough collection of simulation results will be reported on 
later . 



For each of the 12 simulation runs (each column 
represents a run) summarized in Table 3.1, 100 samples of 
approximate size 50 each were used. Each run of 100 simulated 
samples took approximately an hour of computing time on a VAX 
11/780, which explains why more extensive runs were not made. 

The upper part of Table 3.1 contains the number (out of 
100) of calculated approximate 90J confidence intervals which 
actually trapped the true parameter values being estimated. 
Lawless (2) has predicted that these confidence intervals 
would tend to be too large (and hence trap true values more 
than 90 % of the time) for samples with small numbers of 
failures. Our results seem to support that prediction when 
the number of failures is 10 or 20, but such an effect is not 
obvious in the 3 or 5 failure cases. More extensive 
simulations will be needed to gain insight into this matter. 


Eta «* 100 



Beta-1 . 5 


Beta-2 

.0 


Beta-3.0 


Failures 3 

3 

5 

10 

20 

3 

5 

10 

20 

3 

5 

10 

20 

P 

85 

91 

90 

99 

88 

92 

93 

89 

95 

87 

90 

84 

"^.OOI 

93 

89 

93 

99 

88 

90 

93 

88 

88 

90 

91 

92 

t.ol 

93 

89 

93 

98 

87 

92 

90 

89 

91 

90 

91 

o 

ON 

t.| 

87 

91 

91 

96 

92 

92 

. 

94 

91 

91 

95 

90 

92 


86 

91 

88 

^95 

91 

91 

95 

89 

90 

96 

96 

95 

*.5 

84 

91 

84 

95 

89 

86 

93 

95 

90 

92 

97 

97 


85 

90 

85 

96 

88 

86 

95 

94 

90 

91 

96 

96 


88 

91 

90 

95 

88 

90 

94 

92 

91 

91 

94 

96 

a 
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2.03 

1.75 

1.59 

1.58 

2.76 
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2.18 
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3.78 

3.42 

3.25 

3.07 

p too large 

59 

55 

48 

56 

64 

58 

61 

52 

64 

57 

52 

50 

Average n. 
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107 
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99 

99 
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99 
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41 

34 

39 

48 

35 

36 

38 

51 

35 
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45 
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Number of 
Confidence 
Intervals 
Trapping 
Actual Values 


Each column based on 100 samples of (approximate) size 50. 


TABLE 3.1 
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In the bottom part of Table 3.1, the bias in the 
estimators £ and £ is examined, and we see that for smaller 
samples the bias in each estimator grows larger. Both *p andft 
are biased high (in average), but in the case of ft , median 
values are below the actual parameter values. The possibility 
of constructing (by simulation) a table of bias-correcting 
multipliers as functions of sample size, 'p , ^ , and the 

number of failures may be worth considering. 

Although these simulation results are not extensive 
enough to provide a definitive conclusion, the tentative 
conclusion is that the approximate confidence intervals 
calculated using the likelihood ratio statistic are reasonably 
accurate even for samples with very few failures. Thus, the 
calculation of these intervals would seem to be a useful 
addition to the SSME Weibull analysis methodology. 

We should note here that the calculations of the maximum 
likelihood estimators and the associated approximate 
confidence intervals are based on the assumption that our 
failure times are true point data, i.e. that we know the 
failure times exactly. Whenever failure times are known only 
to the extent of falling in some time interval (interval 
data), the above calculations do not apply. To use the above 
methods with interval data, we must make some assumption about 
the placement of failure times within failure intervals. 
Since much of the SSME failure data is interval data, we will 
pursue this issue in the next section. 


4. INTERVAL DATA CONSIDERATIONS. 


The most conservative approach in working with interval 
data is to make no additional assumptions about the true 
failure times and use only the failure intervals in any 
statistical analysis. It is not difficult to modify the 
likelihood function to accomodate interval data, and calculate 
the maximum likelihood estimates for p and w based on this 
function. The numerical procedures are a bit more complicated 
and sensitive than in the point data case; hence reasonably 
good beginning guesses forpand n. are required in order for 
the iterative methods to converge properly. The capability to 
calculate interval data maximum likelihood estimates for p and 
K. has been added to the Weibull analysis computer program 
being used at MSFC. 

One approach which has been used in analyzing SSME 
interval data is to make the optimistic assumption that all 
failures in a failure interval occur at the right hand 
boundary of the interval. As might be expected, the estimates 
for p and h. that are calculated using the interval data 
method and the point data method (with the above assumption) 
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are quite different for most sets of data. Of course the 
differences in these estimates will decrease as the intervals 
involved get smaller, but for the interval sizes typical of 
current SSME data, these differences can be expected to be 
significant . 

Chart 4.1 illustrates a range of possible assumptions 
about failure time placement for interval data. All of these 
choices have been included as options in the Weibull analysis 
computer program. This capability allows an exploration of 
the implications arising from various assumptions about the 
unknown failure times. In the computer calculation for choice 
A (interval data method), the initial "guesses” for p and 
are gotten by calculating estimates using choice D. 


Choice A 


•^t 


-h- 


+ 3 

—I 


4 failures 3 failures 


No assumptions . 


Choice B 


Choice C 


Choice D 



4 failures 
at -t-i. 


*1 

k • \- 

4 failures 
at midpt. 





3 failures 
at +3 


±7 

7 ' 

3 failures 
at midpt . 


Optimistic 
assumption . 


" Neutral" 
assumption #1 


"Neutral" 
assumption //2 


-£-3 

failures equally spaced 
-none at interval endpoints. 


Choice E 


^ 2 - *3 

♦ # 1 

/ t 

4 failures 3 failures 

at -t-. at -fc 

‘ 2 , 


Pessimistic 
assumption . 


CHART 4.1 
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CONCLUSIONS 


The primary conclusion of this study is that Weibull 
methods can be of positive value in SSME hardware reliability 
analysis. However, caution should be exercised in the use of 
these methods with small-failure samples and interval data. 

When using point data, it is important to consider 
confidence intervals because point estimates can be quite 
erroneous and misleading, especially when samples contain very 
small numbers of failures. One of the primary results of this 
study was to implement computer methods for calculating 
approximate confidence intervals from such samples and to 
verify that these approximate confidence intervals are 
reasonably accurate. 

The use of the Weibayes analysis method discussed in (1) 
is especially risky for current SSME use because of the lack 
of a significantly large SSME failure data base. In fact, 
making a priori estimates of betas from the existing SSME 
failure data base is not much better than simple guessing. 

When analyzing interval data, it is important to realize 
that any assumptions made about the placement of the failure 
times within failure intervals is likely to have more impact 
upon the statistical predictions made from that data, than the 
choice of a particular statistical method, or even model, is 
likely to have. This is not to suggest that such assumptions 
should never be made, but rather to point out that when they 
are made, careful consideration and study should be given the 
choice of an assumption. Additions to the Weibull analysis 
computer program allow explorations of the implications of 
various of these assumptions. 

Finally, no matter what assumptions are chosen, it is 
imperative to remember that the predictive power of any model 
is limited by the integrity and informational content of the 
data used as input to that model. The SSME data is typically 
interval data with a small number of failures, and no 
statistical model or method will be capable of extracting 
extremely dependable and accurate predictions from it. 

In summary, the modified Weibull analysis computer 
program now provides a range of capabilities and options for 
treating estimating methods and data assumptions. These 
capabilities enhance the chances that the Weibull model can be 
used to advantage in SSME hardware reliability analysis. When 
applied and interpreted with the caution dictated by the 
considerations outlined here, the model should provide an 
important additional tool for SSME data analysis. 
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